Learning to reach by reinforcement learning using a receptive field based function approximation approach with continuous actions

Reinforcement learning methods can be used in robotics applications especially for specific target-oriented problems, for example the reward-based recalibration of goal directed actions. To this end still relatively large and continuous state-action spaces need to be efficiently handled. The goal of this paper is, thus, to develop a novel, rather simple method which uses reinforcement learning with function approximation in conjunction with different reward-strategies for solving such problems. For the testing of our method, we use a four degree-of-freedom reaching problem in 3D-space simulated by a two-joint robot arm system with two DOF each. Function approximation is based on 4D, overlapping kernels (receptive fields) and the state-action space contains about 10,000 of these. Different types of reward structures are being compared, for example, reward-on- touching-only against reward-on-approach. Furthermore, forbidden joint configurations are punished. A continuous action space is used. In spite of a rather large number of states and the continuous action space these reward/punishment strategies allow the system to find a good solution usually within about 20 trials. The efficiency of our method demonstrated in this test scenario suggests that it might be possible to use it on a real robot for problems where mixed rewards can be defined in situations where other types of learning might be difficult

Prevention and treatment of peri-implant diseases—The EFP S3 level clinical practice guideline

Background: The recently published Clinical Practice Guidelines (CPGs) for the treatment of stages I–IV periodontitis provided evidence-based recommendations for treating periodontitis patients, defined according to the 2018 classification. Peri-implant diseases were also re-defined in the 2018 classification. It is well established that both peri-implant mucositis and peri-implantitis are highly prevalent. In addition, peri-implantitis is particularly challenging to manage and is accompanied by significant morbidity. Aim: To develop an S3 level CPG for the prevention and treatment of peri-implant diseases, focusing on the implementation of interdisciplinary approaches required to prevent the development of peri-implant diseases or their recurrence, and to treat/rehabilitate patients with dental implants following the development of peri-implant diseases. Materials and Methods: This S3 level CPG was developed by the European Federation of Periodontology, following methodological guidance from the Association of Scientific Medical Societies in Germany and the Grading of Recommendations Assessment, Development and Evaluation process. A rigorous and transparent process included synthesis of relevant research in 13 specifically commissioned systematic reviews, evaluation of the quality and strength of evidence, formulation of specific recommendations, and a structured consensus process involving leading experts and a broad base of stakeholders. Results: The S3 level CPG for the prevention and treatment of peri-implant diseases culminated in the recommendation for implementation of various different interventions before, during and after implant placement/loading. Prevention of peri-implant diseases should commence when dental implants are planned, surgically placed and prosthetically loaded. Once the implants are loaded and in function, a supportive peri-implant care programme should be structured, including periodical assessment of peri-implant tissue health. If peri-implant mucositis or peri-implantitis are detected, appropriate treatments for their management must be rendered. Conclusion: The present S3 level CPG informs clinical practice, health systems, policymakers and, indirectly, the public on the available and most effective modalities to maintain healthy peri-implant tissues, and to manage peri-implant diseases, according to the available evidence at the time of publication

Odor supported place cell model and goal navigation in rodents

Experiments with rodents demonstrate that visual cues play an important role in the control of hippocampal place cells and spatial navigation. Nevertheless, rats may also rely on auditory, olfactory and somatosensory stimuli for orientation. It is also known that rats can track odors or self-generated scent marks to find a food source. Here we model odor supported place cells by using a simple feed-forward network and analyze the impact of olfactory cues on place cell formation and spatial navigation. The obtained place cells are used to solve a goal navigation task by a novel mechanism based on self-marking by odor patches combined with a Q-learning algorithm. We also analyze the impact of place cell remapping on goal directed behavior when switching between two environments. We emphasize the importance of olfactory cues in place cell formation and show that the utility of environmental and self-generated olfactory cues, together with a mixed navigation strategy, improves goal directed navigation

Self-influencing synaptic plasticity: recurrent changes of synaptic weights can lead to specific functional properties

Recent experimental results suggest that dendritic and back-propagating spikes can influence synaptic plasticity in different ways (Holthoff, 2004; Holthoff et al., 2005). In this study we investigate how these signals could interact at dendrites in space and time leading to changing plasticity properties at local synapse clusters. Similar to a previous study (Saudargiene et al., 2004) we employ a differential Hebbian learning rule to emulate spike-timing dependent plasticity and investigate how the interaction of dendritic and back-propagating spikes, as the post-synaptic signals, could influence plasticity. Specifically, we will show that local synaptic plasticity driven by spatially confined dendritic spikes can lead to the emergence of synaptic clusters with different properties. If one of these clusters can drive the neuron into spiking, plasticity may change and the now arising global influence of a back-propagating spike can lead to a further segregation of the clusters and possibly the dying-off of some of them leading to more functional specificity. These results suggest that through plasticity being a spatial and temporal local process, the computational properties of dendrites or complete neurons can be substantially augmented

Developing velocity sensitivity in a model neuron by local synaptic plasticity

Sensor neurons, like those in the visual cortex, display specific functional properties, e.g., tuning for the orientation, direction and velocity of a moving stimulus. It is still unclear how these properties arise from the processing of the inputs which converge at a given cell. Specifically, little is known how such properties can develop by ways of synaptic plasticity. In this study we investigate the hypothesis that velocity sensitivity can develop at a neuron from different types of synaptic plasticity at different dendritic sub-structures. Specifically we are implementing spike-timing dependent plasticity at one dendritic branch and conventional long-term potentiation at another branch, both driven by dendritic spikes triggered by moving inputs. In the first part of the study, we show how velocity sensitivity can arise from such a spatially localized difference in the plasticity. In the second part we show how this scenario is augmented by the interaction between dendritic spikes and back-propagating spikes also at different dendritic branches. Recent theoretical (Saudargiene et al. in Neural Comput 16:595-626, 2004) and experimental (Froemke et al. in Nature 434:221-225, 2005) results on spatially localized plasticity suggest that such processes may play a major role in determining how synapses will change depending on their site. The current study suggests that such mechanisms could be used to develop the functional specificities of a neuronVytauto Didžiojo universiteta

Extraction and Investigation of Lane Marker Parameters for Steering Signal Prediction

In the work several lane marker parameters, extracted from the visual data that can be used for steering prediction in the intelligent self-learning driver's assistance systems, are presented. Stable parameters that can be used for steering prediction were estimated from the lane marker. The presented parameters show high correlation with a car steering angle. Lane marker detection and lane marker extraction methods from mono-camera images are also considered in the paper

Vehicle Acceleration Prediction Using Specific Road Curvature Points

In the work vehicle acceleration prediction issue is discussed. Three types of parameters are used for prediction system input: CAN-bus parameters - speed and curvature, derived speed parameters and newly offered specific curve point parameters, denoting changes in a curve. The real road data was used for predictions. Road curvature segments were divided into single and S-type curves. Acceleration was predicted using artificial neural networks and look-up table. The look-up table method showed the best results with newly offered specific curve parameters

Vehicle Acceleration Prediction Using Specific Road Curvature Points

In the work vehicle acceleration prediction issue is discussed. Three types of parameters are used for prediction system input: CAN-bus parameters - speed and curvature, derived speed parameters and newly offered specific curve point parameters, denoting changes in a curve. The real road data was used for predictions. Road curvature segments were divided into single and S-type curves. Acceleration was predicted using artificial neural networks and look-up table. The look-up table method showed the best results with newly offered specific curve parameters